Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

These commit is mostly resolve the memory leak problem and optimize memory usage to handle millions of requests #81

Open
wants to merge 17 commits into
base: master
Choose a base branch
from

Conversation

truong-hua
Copy link
Contributor

These commits optimize memory allocation to reuse buffer instead of keep allocating and release the memory on every request which push a lot of pressure on GC.

This pull request also add a notice and an config option to prevent memory leak if client sending Authorization while Pulsar broker does not activate it. And an option to keeps both Pulsar token authorization and normal HTTP Authorization without confliction.

This pull request also allow to set a limit on number of concurrency (config WorkerPoolSize) to limit number of concurrent request to protect the brokers as same as leverage reusable memory buffer. Without this one, an accidentally peak time can cause crashing of service and uncontrolable of resource usage.

This pull request is tested and using on production environment of https://doopage.com (we are using only Pulsar receiver) to handle few millions requests per day and it definitely improve the performance by using less CPU (mostly spend on GC) and of course much less Memory.

…t that contains both HTTP headers and body as a single message
… pressure on GC which reduce a lot of CPU usage and memory usage when receive hugh amount number of request
…h is very useful to allow user to disable a config
…mmit prevent confiction when using Authorization header for both JWT and Pulsar token and may be other authorization purpose.

Without this commit, using Authorization header as Pulsar token may cause memory leak because of token change per every request cause pulsar-beam to create a new pulsar client every time.
With configurable Authorization, user can also use ?includeHeader to get Authorization header to validate a payload without casuing pulsar-beam memory leak
…Request Line in the payload to send to Pulsar, which is useful in case of Pulsar consumer would like to detect the url or method that webhook client is using
…ation with a reverse proxy engine in front of or using other Authorization method.

This commit also add a short brief to let people know how to detect memory leak and resolve it. Further default configuration of Pulsar Beam should prevent this case from happening.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant